feat: add cloud cleanup script for stale integration test artifacts #798

aaronsteers · 2025-09-19T21:03:08Z

This PR targets the following PR:

feat: add pagination support and name attribute for cloud objects: CloudConnection, CloudSource, and CloudDestination #782

feat: add cloud cleanup script for stale integration test artifacts

Summary

Adds a helper script bin/cleanup_cloud_artifacts.py to clean up stale connector and connection definitions left behind by cloud integration tests. The script:

🔍 Scans source code for hardcoded UUIDs using regex and exempts them from cleanup
📋 Lists connections, sources, and destinations using existing PyAirbyte cloud operations
🛡️ Defaults to --dry-run mode for safety, requires --no-dry-run for actual deletions
🔗 Prints web URLs for resources that would be deleted in dry-run mode
⚙️ Uses existing authentication patterns via environment variables
🚀 Includes poe cloud-cleanup task for easy execution

Review & Testing Checklist for Human (4 items)

Test end-to-end with real cloud credentials - This is critical since I couldn't test actual cloud operations due to missing credentials
Verify UUID exemption logic works correctly - Test that hardcoded UUIDs in the codebase are actually skipped during deletion
Test both dry-run and actual deletion modes - Ensure dry-run doesn't accidentally delete anything and that --no-dry-run works as expected
Verify error handling with invalid credentials/network issues - Test graceful failure scenarios

Test Plan

Run poe cloud-cleanup (dry-run mode) with valid cloud credentials to see what would be deleted
Verify that known hardcoded UUIDs from test files are listed as exempted
Create a test connector/connection and verify it appears in the deletion list
Test poe cloud-cleanup --no-dry-run on the test resources to verify actual deletion works

Notes

Script successfully scans source code and found 11 hardcoded UUIDs during local testing
All linting, formatting, and type checking passes
CLI interface works correctly with --help, --dry-run, and --no-dry-run flags
Could not test actual cloud workspace operations due to missing credentials in dev environment

Session: https://app.devin.ai/sessions/b580c93f6f144ca8b141dce686201892
Requested by: @aaronsteers

- Add limit, offset, and include_deleted parameters to list_connections, list_sources, and list_destinations - Update underlying api_util functions to accept and pass through pagination parameters - Maintain backward compatibility with optional parameters and sensible defaults - Update docstrings to document new pagination parameters - Add noqa comments to suppress PLR0913 linting warnings for necessary parameter count Co-Authored-By: AJ Steers <[email protected]>

…kspace

- Scans source code for hardcoded UUIDs to exempt from cleanup - Lists and deletes stale connections, sources, and destinations - Defaults to dry-run mode for safety - Uses existing PyAirbyte cloud operations and authentication - Includes poe task for easy execution Co-Authored-By: AJ Steers <[email protected]>

devin-ai-integration · 2025-09-19T21:03:12Z

Original prompt from AJ Steers

@Devin - Create a PR on top of this one: <https://github.com/airbytehq/PyAirbyte/pull/782/files>
In the new PR, create a helper script in the 'bin' directory of pyairbyte. The goal of the helper script is to list and delete connector definitions and connection definitions that are stale artifacts from the cloud integration tests in the same repo. The script should start by scanning the source code to find hard-coded connection GUIDs which might be connection IDs or connector IDs. You can just use regex for GUID/UUID detection and no other smarts here. That hard coded list will be compared against any GUIDs/UUIDs that we see, and any matches will be exempted from the cleanup. The script should default to --dry-run=true and only actually do deletions if --dry-run=false. Use the existing 'list' operations to list connections and connectors (sources and destinations) and use the "permently_delete()' methods to perform the deletions. In dry-run mode, we'll just print links to everything that _would_ be deleted, including the name of the object that would be deleted.
Thread URL: https://airbytehq-team.slack.com/archives/D089P0UPVT4/p1758315045352879?thread_ts=1758315045.352879

devin-ai-integration · 2025-09-19T21:03:13Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2025-09-19T21:03:30Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This PyAirbyte Version

You can test this version of PyAirbyte using the following:

# Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1758315185-cloud-cleanup-script' pyairbyte --help

# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1758315185-cloud-cleanup-script'

Helpful Resources

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

/fix-pr - Fixes most formatting and linting issues
/poetry-lock - Updates poetry.lock file
/test-pr - Runs tests with the updated PyAirbyte

Community Support

Questions? Join the #pyairbyte channel in our Slack workspace.

📝 Edit this welcome message.

github-actions · 2025-09-19T21:09:08Z

PyTest Results (Fast Tests Only, No Creds)

301 tests ±0 301 ✅ ±0 4m 24s ⏱️ -19s
1 suites ±0 0 💤 ±0
1 files ±0 0 ❌ ±0

Results for commit 1e0fcf2. ± Comparison against base commit 31c8f15.

♻️ This comment has been updated with latest results.

- Regenerated poetry.lock to match current pyproject.toml changes - Resolves Poetry version conflict (2.2.0 vs 2.0.1) - Fixes 'pyproject.toml changed significantly' error - All imports and script functionality verified locally Co-Authored-By: AJ Steers <[email protected]>

- Removed direct six dependency (DEP002 violation) - Six still available as transitive dependency via python-dateutil - Verified locally: deptry check passes and script works correctly Co-Authored-By: AJ Steers <[email protected]>

github-actions · 2025-09-19T23:04:18Z

PyTest Results (Full)

364 tests ±0 346 ✅ ±0 21m 10s ⏱️ +32s
1 suites ±0 16 💤 ±0
1 files ±0 2 ❌ ±0

For more details on these failures, see this check.

Results for commit 1e0fcf2. ± Comparison against base commit 31c8f15.

♻️ This comment has been updated with latest results.

…kspace

…o devin/1758315185-cloud-cleanup-script

devin-ai-integration bot and others added 8 commits September 11, 2025 02:17

clean up and add real pagination

18d25d4

Auto-fix lint and format issues

4f1a82e

fix 'next' refs

5836e1a

Apply suggestion from @aaronsteers

784f024

Auto-fix lint and format issues

55c06bf

Merge branch 'main' into devin/1757556687-add-pagination-to-cloud-wor…

7f442f4

…kspace

devin-ai-integration bot assigned aaronsteers Sep 19, 2025

devin-ai-integration bot and others added 2 commits September 19, 2025 21:55

aaronsteers added 6 commits September 19, 2025 19:05

(cherry-pick-me) fix: fix pagination and add a name field

8084f87

(cherry-pick-me) fix: fix pagination and add a name field

ca38695

Merge branch 'main' into devin/1757556687-add-pagination-to-cloud-wor…

31c8f15

…kspace

fix script logic

6ef5959

Merge branch 'devin/1757556687-add-pagination-to-cloud-workspace' int…

dfcbc0e

…o devin/1758315185-cloud-cleanup-script

poetry lock

1e0fcf2

Base automatically changed from devin/1757556687-add-pagination-to-cloud-workspace to main September 20, 2025 03:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add cloud cleanup script for stale integration test artifacts #798

feat: add cloud cleanup script for stale integration test artifacts #798

Uh oh!

aaronsteers commented Sep 19, 2025

Uh oh!

devin-ai-integration bot commented Sep 19, 2025

Uh oh!

devin-ai-integration bot commented Sep 19, 2025

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

github-actions bot commented Sep 19, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 19, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

feat: add cloud cleanup script for stale integration test artifacts #798

Are you sure you want to change the base?

feat: add cloud cleanup script for stale integration test artifacts #798

Uh oh!

Conversation

aaronsteers commented Sep 19, 2025

feat: add cloud cleanup script for stale integration test artifacts

Summary

Review & Testing Checklist for Human (4 items)

Test Plan

Notes

Uh oh!

devin-ai-integration bot commented Sep 19, 2025

Uh oh!

devin-ai-integration bot commented Sep 19, 2025

🤖 Devin AI Engineer

Uh oh!

github-actions bot commented Sep 19, 2025

👋 Greetings, Airbyte Team Member!

Testing This PyAirbyte Version

Helpful Resources

PR Slash Commands

Community Support

Uh oh!

github-actions bot commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Fast Tests Only, No Creds)

Uh oh!

github-actions bot commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PyTest Results (Full)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions bot commented Sep 19, 2025 •

edited

Loading

github-actions bot commented Sep 19, 2025 •

edited

Loading